A. Field of the Invention
This invention relates generally to hyperspectral and mutispectral imaging systems for aerial reconnaissance and surveillance applications using a wafer-scale Focal Plane Array (FPA) of an area array format. In this art, xe2x80x9chyperspectralxe2x80x9d imaging systems refer to those in which radiation at thirty or more discrete wavelengths are imaged. Imagers that image a lesser but plural number of spectral bands are referred to as xe2x80x9cmultispectralxe2x80x9d imagers. Such systems are used in various applications, including Department of Defense satellite systems and commercial land resource management imaging systems.
The invention further relates to a scanning method for directing a spectrally separated image of the object under surveillance over the FPA in a predetermined direction and rate. In another aspect, the invention further relates to a time delay and integrate (TDI) clocking technique in which the FPA is separated into discrete sub-arrays, in which the length of TDI clocking within each sub-array is predetermined according to the spectral responsiveness of the detector material to each spectral band detected by each sub-array. The present invention provides improvement over the prior art by significantly increasing the area of coverage scanned by a multispectral or hyperspectral imaging system.
B. Description of Related Art
Hyperspectral imaging is a well established art based on the science of spectroradiometers and spectral imaging. In hyperspectral imaging, image radiation is separated into its spectral components using optical elements such as filters, prisms or diffraction gratings, and the separated radiation is imaged by a detector. This section will briefly review the state of the art and cite to several references and patents discussing background art. The entire content of each of the references and patents cited in this document is incorporated by reference herein.
Discussion of the basic hyperspectral imaging technology can be found in such references as The Infra-Red and Electro-Optical Handbook, Vol. 1, George J. Zissis, Editor, Sources of Radiation, pp. 334-347. Spectroradiometers applied to reconnaissance applications are called imaging spectrometers, wherein each image is collected in a number of spectral wavelengths.
The fundamental design of imaging spectrometers can be traced to Breckinridge, et. al., U.S. Pat. No. 4,497,540. In this patent, a scene is imaged on slit and dispersed using a diffraction grating or prism to illuminate a detector array. All spectral channels are acquired simultaneously and the field of view in each channel is precisely the same. This can be contrasted to xe2x80x9cspectral imagingxe2x80x9d, in which different spectral channels are acquired sequentially in time, i.e. through filters, and are not inherently spatially registered to one another.
The common elements of a hyperspectral imaging spectrometer system are shown in FIG. 1. Target imagery is projected through a fore-optic 10 into the aperture slit 12 of a spectrometer system 14. The spectrometer 14 includes an optical grating 16, which serves to diffract the image into its spectral elements. The diffracted image is imaged by camera optics 18 onto a focal plane array detector 20. Each image 22 generated by detector extends in one spatial direction, typically across the line of flight. A slice of the image in that spatial direction is imaged in a large number (thirty or more) different bands of the spectrum. As the platform flies pasts the terrain of interest (or as different terrain is scanned onto the imager with a scanning device) successive images 22 are generated, each in one spatial direction. When the series of images are combined, they yield an xe2x80x9cimage cubexe2x80x9d 24, covering the terrain of interest in two orthogonal directions in a multitude of spectra.
The width of the slit 12 is directly proportional to the amount of radiation, and therefore the signal-to-noise-ratio, of the system. And, it is inversely proportional to the spatial resolution. That is, the wider the slit, the worse the spatial resolving power of the system. On the other hand, the spectral resolution of the system is dependent on the optical parameters of the diffraction grating and the number of pixel elements of the detector. The higher the quality (blaze) of the diffraction grating 16 and the greater the number of pixels on the focal plane 20, the higher the spectral resolution.
In a typical operation, the hyperspectral system is moved over the scene of interest either by moving the entire system (i.e. xe2x80x9cpushbroomxe2x80x9d) or by movement of a scan mirror (not shown). As objects enter the field of view of the aperture slit 12, they are diffracted into their spectral colors and projected through the camera lens 18 to the detector array 20. Thus, each line of the target area is projected as a two dimensional image onto the detector, with one dimension representing a spatial axis of the image in-line-of-flight (ILOF) or cross-line-of-flight (XLOF), depending on camera orientation and scanning mode, and the other dimension representing the spectrum of the image. The combination of the scanning action of the camera, either by pushbroom or scan mirror, with the frame rate of the focal plane array, determines the number of xe2x80x9cframesxe2x80x9d of the resulting image.
A number of frames of the image will provide spatial resolution in the scan direction. Typically, a design will call for roughly equal resolution in both directions. In this fashion, an image xe2x80x9ccubexe2x80x9d 24 is thus created wherein the hyperspectral imagery is subsequently processed to detect object features of interest to the image analyst. There exists a tremendous body of knowledge regarding the processing of hyperspectral imagery and how it applies to target xe2x80x9cphenomenologyxe2x80x9d, which is outside the scope of this disclosure. For up to date background information on the processing of hyperspectral imagery see, for example, SPIE Proceedings, Vol. 3372, Algorithms for Multispectral and Hyperspectral Imagery IV, Apr. 13-14, 1998.
A common example of a hyperspectral imaging spectrometer is the NASA HIRIS instrument, which is designed to image the earth""s surface from an 800-km altitude. The earth""s surface is focused on a narrow slit and then dispersed and refocused onto a two dimensional focal plane array. Each time the satellite progresses around the earth by the equivalent of one pixel, the focal plane is read out and one frame of the image xe2x80x9ccubexe2x80x9d is created. It is commonly recognized that systems such as HIRIS have the limitations of limited resolution, limited exposure time and spatial smearing. There exists a large open literature which discusses technological issues pertaining to hyperspectral imaging systems, major development programs and existing national and commercially available systems. See, for example, Kristin Lewotsky, xe2x80x9cHyperspectral imaging: evolution of imaging spectrometryxe2x80x9d, OE Reports, November 1994. For a comprehensive reference list of imaging spectrometers, see Herbert J. Kramer, Observation of the Earth and Its Environmentxe2x80x94Survey of Missions and Sensors, Second Edition, Springer-Verlag, 1994.
Spectroscopy using a large format, two-dimensional CCD in a scanning configuration is disclosed in Bilhorn, U.S. Pat. No. 5,173,748 entitled Scanning Multi-Channel Spectrometry using a Charge-Coupled Device (CCD) in Time-Delay Integration (TDI) Mode. Bilhorn discloses a technique in which the performance of a spectrometer is improved by scanning the dispersed spectrum across a Time Delay and Integrate (TDI) CCD array synchronously with the clocking of the charge within the array. This technique in Bilhorn affords an accumulation of signal charge from each wavelength of the spectrum, which eliminates the need to compensate for throughput variations commonly found when using other detector approaches such as photodiode arrays, and increases the overall response of the spectrometer. This invention was designed for the use of determining the spectrum of a polychromatic light source in a laboratory environment where a slit is adjusted to achieve the best compromise between light throughput and spectral resolution.
U.S. Pat. No. 5,379,065 to Cutts discloses a method of performing spectral imaging in low earth orbit with rapid image motion, without the limitations identified in the above example, by the use of a spectrally agile filter using Acusto Optic Tunable Filtering (AOTF) technology. The technique relies on the use of a fixed slit, the ability of the AOTF to shutter the FPA and motion compensation accomplished by synchronizing movement of charges located within the FPA with the moving image during the exposure time. The invention synchronizes changes in spectral pass band with frame transfers of the focal plane array imager. Different spectral bands are given different exposure times to maximize the signal to noise across the entire spectrum.
U.S. Pat. No. 5,760,899 by Eismann, entitled High-Sensitivity Multispectral Sensor, discloses the use of a dispersive spectrometer and filtered TDI detector to improve dwell time, temporal simultaneity and spatial registration. In his preferred embodiment, however, Eismann does not use TDI in the conventional manner. Rather, Eismann discloses TDI segments which are formed from a discrete set of non-contiguous, equally spaced field angles, with TDI preferably achieved off-chip; that is, the array is read out in frames and the properly shifted rows are digitally summed on a frame-to-frame basis. He later describes this as xe2x80x9cframe averagingxe2x80x9d (Col. 6, Line 57). The benefit achieved from his approach is the fact that all spectra of interest are collected simultaneously, avoiding a temporal delay and therefore distortion in the resulting image cube. As a byproduct of his spectral imaging technique, Eismann must reduce the velocity of the platform carrying the hyperspectral sensor in order to achieve a sufficient increase in signal performance (Col. 5, Line 50-52). This is highly undesirable and unusable for applications in high-speed reconnaissance aircraft, particularly in a military reconnaissance situation.
Further background art exists in the field of wafer scale FPA processing as related to airborne tactical reconnaissance applications. See, for example, Lareau, et al., U.S. Pat. No. 5,155,597, which describes an wafer-scale electro-optical imaging array. Wafer-scale focal plane arrays are now well known in the art and information describing their design and use in reconnaissance applications can be found at several places in the literature, including: S. J. Strunk, J. McMacken, S. Kamasz, W. Washkurak, F. Ma, and S. G. Chamberlain, xe2x80x9cThe Development of a 4 Million Pixel CCD Imager for Aerial Reconnaissancexe2x80x9d, SPIE Vol. 1763, Airborne Reconnaissance XVI, July 1992, pp. 25-36; M. Farrier, S. R. Kamasz, F. Ma, W. D. Washkurak, S. G. Chamberlain and P. T. Jenkins, xe2x80x9cMegapixel Image Sensors with Forward Motion Compensation for Aerial Reconnaissance Applicationsxe2x80x9d, SPIE Vo. 2023, Airborne Reconnaissance XVII, 12-14 Jul. 1993; S. G. Chamberlain, et. al., xe2x80x9c25 Million pixel CCD image sensor,xe2x80x9d SPIE Proceedings, Vol. 1900, Charge-Coupled Devices and Solid State Optical Sensors III, 2-3 Feb. 1993; and, A. G. Lareau, xe2x80x9cElectro-Optical imaging array with motion compensation,xe2x80x9d SPIE Proceedings, Vol. 2023, Airborne Reconnaissance XVII, 12-14 Jul. 1993, pp. 65-79. In the preferred embodiments of the present invention, the focal plane array is as large as possible, preferable being of xe2x80x9cwafer-scalexe2x80x9d, implying a size of at least 60-mm on a side for 4-inch wafer processing or 80-mm on a side for 6-inch wafer processing.
Still further background art exists regarding the Quantum Efficiency (QE) and responsivity of semiconductor detector devices and the TDI technique used to improve the SNR within detector devices. It is well known within the art of developing imaging Focal Plane Arrays that the QE and responsiveness of the detector material varies according to the wavelength (frequency) of the light energy which it is detecting. For example, information on the spectral response characteristics of the silicon in regards to FPA processing can be found in S. M. Sze, Physics of Semiconductor Devices, Ch. 7, John Wiley and Sons, Inc. (1981). To increase the signal level for images with overall low illumination or to improve signal levels for images with spectral content for which the array has a poor responsivity, it is common to use a TDI technique with some type of a scanning mechanism. TDI techniques are again well known in the art. Background information for that technique can be found, for example, in the patent to William R. Pfister, U.S. Pat. No. 5,231,502.
Closer to the field of airborne tactical reconnaissance, another recent development has been publicly released by the United State Navy""s Naval Research Laboratory called Dark Horse 1. It is known as the xe2x80x9cAutonomous Airborne Hyperspectral Imaging and Cueing Systemxe2x80x9d, and was flown aboard a P-3 aircraft at the Patuxent River Naval Air Station, Lexington Park, Md., in 1999. It uses a grating type spectrometer with a 50-xcexcm slit to project 64 bands within the 400-850 nm spectrum onto a 512xc3x97512 pixel FPA at 100 frames per second. The FPA pixel size is 15-xcexcm which yields a 1.3-m resolution XLOF from 7100-ft vertical through a 50-mm lens with 2:1 pixel binning. The Dark Horse camera was used to cue a high resolution tactical reconnaissance camera, the CA-260, manufactured by Recon/Optical, Inc. with a 12-in focal length and a 12-xcexcm pixel pitch, 5000xc3x975000 pixel, wafer scale FPA. Although the results of the demonstration were considered a great success, they also served to emphasize the severe limitation of modern hyperspectral sensors in terms of area coverage. As flown, the Dark Horse sensor demonstrated a coverage capability of only 34.8-nmi2/hr, which is far below the existing capability of high resolution monochromatic coverage rates of 8000 to 10000- nmi2/hr. And further, even at that reduced coverage rate, the sensor achieved resolutions of 4.2-m in the in-line of flight direction (ILOF) and 1.3-m in the cross-line of flight (XLOF) which is worse than the 1-m resolution stated goal. More details about this system can be obtained from the Naval Research Laboratories, Optical Sciences Division, Advanced Concepts Branch, Code 5620, Washington, D.C. 20375.
So, it is easy to see that to be effective as a cueing sensor, a dramatic increase in coverage capability and resolution of hyperspectral sensors is needed. The hyperspectral FPA of the present invention meets these needs by providing both a significant increase in coverage capability, and high resolution.
In one aspect, a hyperspectral system is provided based on an electro-optical imaging array arranged as a plurality of rows and columns of individual pixel elements. The array is organized into a plurality of sub-arrays of rows of pixels, each sub-array being responsive to incident radiation from a scene of interest. A multi-spectral or hyper-spectral filter is placed in registry with the electro-optical imaging array. The filter defines a plurality of individual filter bands arranged in optical registry with the sub-arrays whereby each of the sub-arrays receives radiation passing through one of the individual filter bands. A scanning device is provided directing radiation from the scene of interest onto the imaging array. The array and scanning device constructed and arranged such that as the scene of interest is scanned over the array a given point in the scene of interest is sequentially imaged by each of the sub-arrays. In other words, various parts of the scene are imaged simultaneously in different spectral bands but as the scene is scanned each point in the scene is imaged in every spectral band. Clocking circuitry is provided to accomplish movement of charges among the individual pixel elements within each of the sub-arrays in a rate and direction synchronous with the image motion. This allows for increased signal collection, thereby improving the signal to noise ratio of imagery from the array. CMOS and/or CCD implementations are expressly contemplated as possible architectures for the array.
Various novel aspects of the invention include, but are not limited to, the use of an area array FPA in a scanning configuration for simultaneous imaging of many spectral bands. Further, the area array FPA is arranged into the pre-determined number of contiguous sub-arrays which each correspond to a spectral band of the target object. The clocking techniques, which in the illustrated embodiment take the form of TDI techniques, are used as a clocking mechanism within the sub-arrays to increase the SNR of the detected image. Additionally, in one possible embodiment, the TDI length (i.e., number of rows of integration during the exposure) within each sub-array is adjustable to optimize the material response to each spectral band. The array preferably provides for parallel and simultaneous readout of each sub-array to increase the collection rate of the spectral imagery.
In another aspect, a method is provided for obtaining images of a scene of interest in multiple portions of the electromagnetic spectrum with a hyperspectral or multi-spectral imaging system aboard a reconnaissance vehicle. The method comprising the steps of exposing a two-dimensional electro-optical array to scene. The array is arranged as a plurality of rows and columns of individual pixel elements, with the array organized into a plurality of sub-arrays of rows of said pixels. The method continues with the step of controlling the wavelength of the radiation impinging on each of the sub-arrays wherein each sub-array images a different band of the electromagnetic spectrum while the array is exposed. The sub-arrays are exposed to the scene of interest at the same time, with each of the sub-arrays imaging a different portion of the scene of interest simultaneously in a particular band of the spectrum. The method further continues with the step of operating a scanning device for the array so as to scan across the scene of interest while the array is exposed to the scene to thereby direct adjacent portions of the scene of interest onto the imaging array. While the imaging array is exposed to the scene, individual pixel elements of the array are clocked in synchronism with image motion occurring in a direction and at a rate to thereby increase the signal to noise ratio of the collected signal.
Additionally, in one possible embodiment, the TDI length (i.e., number of rows of integration during the exposure) within each sub-array is adjustable to optimize the material response to each spectral band. The array provides for parallel and simultaneous readout of each sub-array to increase the collection rate of the spectral imagery. These features serve to provide a substantial improvement over prior art by dramatically increasing the area imaged per unit of time by a hyperspectral or multispectral imaging system while at the same time increasing the signal to noise ratio of the detected spectral image.